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APPLICATION FOR T.KTTPPQ PATBTfT 
VIDEQ-BASED ST JRGTCAT. TAT?r. KmrO RYSTRM 

Field Of The TnvMififtn 

This invention relates to singical systems in general, and more particularly 
to surgical targeting systems. 

Background Of The Inventiftn 

Many medical procedures must be canied out at an interior site which is 
normally hidden from the view of the physician. In these situations, the 
ph/sidan typically uses some sort of scanning device to examine the patienf s 
anatonty at die intoior site prior to, and in preparation for, conducting die 
actual i»ocedure itself These scanning devices typically inchideNfRI devices, 
CT scanners, machines, ultrasound devices and the like, and serve to 
provide die physician widi a i»eliminaiy knowledge of die patient's internal 
anatomical structure prior to commencing the procedure. The physician can 
dien use diis information to plan die procedure in advance, taking into account 
patient-specific anatomical structure. In addition, die physician can also use 
die information obtained from diese scans to more precisely idmtify die 
tocation of selected structures (eg. tumors and die like) whidh may diemselves 
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be located vnHidn internal organs or otfier mtemal body structures. As a 
result, die physician can more easily **zero in" on such selected structures 
during the subsequent procedure, with less trauma to host organs or other 
internal body structures. Furthermore, in many cases Hit structures of interest 
may be quite small and difficult to identify witfi die naked eye. In these cases, 
preliminary scanning of the patient's internal anatomy using high resolution 
scanning devices can help the physician locate such structures during the 
subsequent procedure. 

In general, scanning devices of the sort described above tend to generate 
a series of two-dimensional (Le., "2-0") images of the patient's anatomical 
stracture. For example, CT scanners generate 2-D images which are viewed 
directly by the physician. By viewing a plurality ofithese 2-D images, the^ 
physician can mentally generate a three-dimensional (ie., "3-D") sense of the 
patient's anatomical structure. 

Some scanning devices create large numbers of 2-D images during die 
scanning process, with each 2-D image representing a plane or slice taken 
dirough the scanned stracture. Furdiermore, some scanning devices also have 
associated computer hardware and software for building a 3-D computer 
model of die patienfs scanned structure using a plurality of these 2-D images. 
For ^cample, scmie MRI devices and some CT scanners have such associated 
computer hardware and software. In diese cases, an operator using diis 
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scanhmg equipment and associated computer haidwaxe and software can 
oeate a nnmber of 2-D images, assemble than into a 3-D cpmputer model of 
the scanned stracture, and then generate various images of that 3*D computer 
model as seen from various angles so as to enhance understanding of the 
patient's anatomical structure. 

While die information generated by the aforementioned scanning devices 
is generally of great benefit to physicians, certain significant limitations still 
exist. For one thing, it can be veiy difficult for a physician to physically 
locate a particular anatomical stnicture during a medical procedure, even when 
that structure is readily identifiable in a scanned image. This may be because 
die structure is very small and difficult to see with the naked eye, or because 
it is not readily visible against a particular backgroundj or because it is itself ^ 
located within an internal organ or other internal body stnicture, etc. For 
anoAer ihirig, even vihen the structure of interest is successfiilly located by die 
physician, it can sometimes still be extranely difficult for die physician to 
reliably direct a medical instniment to diat structure. This may be because die 
structure is quite small and difficult to target accurately, or because 
intervening body structure mandates a complex approach, etc. 
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Ql^ects Of The Invention 

Accordingly, one object of the present invention is to provide a surgical 
taigeting system to facilitate locating a particular anatomical strocture during 
a medical procedure* 

Another object of die present invention is to provide a video-based surgical 
taigedng system to facilitate locating a particular anatomical stnicture during 
a medical procedure. 

And anodier object of the present invention is to provide a video*based 
surgical taigeting system which permits a series of patient-specific 2-D 
images (obtained by scanning patient anatomy using one or more scannmg 
devices of the type desoibed above) to be assembled into a 3-D computer 
model of the patient's scanned stnicture. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which allows a physician to view the aforementioned 
patient-specific 2-D images on a display in any desired access sequence. 

And anodier object of the present invention is to provide a video-based 
surgical targeting system which allows a physician to assemble a series of 
patient-specific 2-D images into a patient-specific database, and dien to 
generate virtual images from the aforementioned patient-specific database, as 
seen from any desired virtual camera position, for viewing on a display. 

Yet another object of the present invention is to provide a video-based 
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surgical taigdiiig system vdiich allows a physician to generate virtual images 
from die aforementioned 3»D computer model, as seen from any desired 
virtual camera position, for viewing on a display. 

And anodier object of die present invention is to provide a video-based 
suigical targeting system ^lich permits a physician to place virtual planning 
markers about any sites of interest while viewing one or more of the 
aforementioned patient-specific 2-D images, with those virtual planning 
maricers then being incorporated into the 3-D computer model, whereby those 
virtual planning markers can be displayed in their appropriate 3-D positions 
when genmting virtual images of the 3-D computer model. 

Still anotiier object of die present invention is to provide a video-based ^ 
surgical targeting system which permits a physician to place virtual planning 
markers about any sites of interest while viewing virtual images of the 3-D 
ccmqmtor model, wifli diose virtual pkmning mark«s then being incorporated 
into the 3-D computer model, whmby those virtual planning markers can be 
displayed in their appropriate positions ^en gmerating subsequent virtual 
images of the 3-D computer model. 

Still another object of the present invention is to provide a video-based 
suigical targeting system whidi permits a physician to place virtual planning 
maricers about any sites of interest while viewing virtual images of the 3-D 
computer model, with diose virtual planning maricers then being incoiporated 
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into tiie 3-D computer model and into the database of 2-D images, whereby 
those \iitiial planning maiters can be displayed in their appropriate positions 
when subsequently generating virtual images of the 3-D computer model or 
when subsequendy dispkiying 2-D images from the patient-specific database. 

Yet another object of die present invention is to provide a video-based 
surgical taigeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be displayed to a physician, and which 
permits a virtual image generated from die 3-D computer model to be 
displayed to a physician, according to the directive of the physician. 

Anodier object of die present invention is to provide a video-based surgical 
taigeting system which permits a real image obtained by a real-time imaging 
device (e.g. a video camera) to be appropriately merged widi a corresponding 
virtual image generated from the 3-D computer model. 

And another object of the present invention is to provide a video-based 
surgical taigeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be qipropriately merged with a 
corresponding virtual image generated from the 3-D computer model, whereby 
die two images will be in registration with one another. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imagmg device (e.g. a video camera) to be merged with a corresponding 
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virtual image generated &om the 3-D computer model, whereby die two 
images will be in registration widi one anoAer, and whereby the physician can 
choose to display either one of the two images exclusive of the other, or a 
composite of bodi images simultaneously. 

Yet another object of the present invention is to provide a video-based 
suigical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be merged with a corresponding 
virtual image generated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
whereby die physician can modify the virtual image gennated from die 3-D 
computer model as needed, by clipping or fading, so as to expose the virtual 
planning markers to view, with the virtual planning markers being 
superimposed on the real image generated by die real-time viewing device. 

And anodic object of the present invention is to provide a video-based 
suigical targeting system which pemiits a real image generated by a real-time 
imaging device (e.g. a video camera) to be merged widi a corresponding 
virtual image generated from the 3-D computer model, whereby die two 
images will be simultaneously displ^ed in registration with one anodier, and 
whereby die physician can modify die virtual image generated from the 3-D 
computer model so as to expose only the virtual planning markers to view, 
with the virtual planning markers being superimposed on the real image 
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generated by tiie real-time imaging device. 

Still another object of the present invention is to provide a video-based 
smgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be merged with a corresponding 
virtual image genoated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
whereby diis registration will be automatically maintained even as the 
real-time imaging device is moved about relative to the anatomical site, with 
tiie virtual image being automatically generated so as to follow the real image. 

Still anotfier object of the present invention is to provide a video-based w 
surgical targeting system which permits a real image obtained by a real-time ^ 
imaging device (e.g. a video camera) to be merged with a corresponding 
virtual image genqated from die 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
v^ieieby diis registration will be automatically maintained through the use of 
a computerized position and orientation tracker connected to the imaging 
device even as die real-time imaging device is moved about relative to die 
anatomical site, with the virtual image being automatically generated so as to 
follow the real image. 

Still another object of die present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
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imaging device (e.g. a video camera) to be meiged with a conesponding 
virtual image generated from die 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
ifidiereby diis registration will be automatically maintained through the use of 
a conqmter search algorithm based on die real inuige and die virtual image 
even as die real-time imaging device is moved about relative to die anatomical 
site, widi the virtual image being automatically generated so as to follow the 
real image. 

Yet another object of the present invention is to provide a video«*based 
surgical targeting system vAdch permits a real image obtained by a reaUtime 
imaging device (e.g. a video camera) to be merged widi a corresponding 
virtual image generated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
vriiereby a smgical instrumoit can be tradced along widi the real-time imaging 
device in order diat die surgical installment can be guided about die anatomical 
site using die virtual image generated from the 3-D computer model. 

And anodier object of die present invention is to provide a video-based 
suigical targeting ^stem which permits a virtual image to be generated from 
the 3-D computer model, and furdier wherein dus virtual image can 
correspond to an ^over die shoulder" view of die working tip of a surgical 
instrument located at die anatomical site. . 



SUBSnTUTE SHEET (RULE 26) 



wo 96^0949 



PCT/US95/13353 



•10- 



And anodier object of the present invntion is to provide an improved 
mediod for locating anatomical stractures during a medical procedure. 

RiimmflTy Of The Invention 

These and other objects of the present invention are achieved through die 
provision and use of a novel video-based surgical targeting system. 

In one form of tiie invention, the video-based surgical targeting system 
conqirises a patient-specific database comprising a plurality of 2-D images of 
the anatomical structure of a patient; a patient-specific 3-D computer model 
of the anatomical structure of the patient, the patient-specific 3-D computer 
model being assmbled fiom the plurality of 2-D images contained in the 
patient-specific database; marker placement means for (i) inserting virtual 
planning mariners into the 2-D images contained in die patient-specific 
database^ and/or (ii) adjusting the positions of virtual planning markers 
inserted into the 2-D images contained in tfie patient-specific database and 
thereafter incorporated into tiiie patient-specific 3-D computer model, or 
inserting virtual planning markers into the 3-D computer model; an image 
generator for generating a virtual image of the anatomical structure modeled 
by the patient-specific 3-D computo: model; real-time image generating means 
for generating a real image of the anatomical structure of a patient video 
mixing means for mixing the virtual image and the real image into an output 
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imag^ whereby tfie ou^t image may comprise eitfaor one of the two images 
exclusive of the other, or a composite of both images; and display means for 
displaying the output image. 

Preferably die video-based surgical targeting system also includes 
registration means for placing tfie virtual image in registration with the real 
image. Theseregistrationmeansmi^includemeansformanually aligning the 
virtual image with the real image, or means for automatically aligning die 
virtual image with the real image. The registration means may also include 
tracking means for tracking Ihe position of die real-time image generating 
means. 

Preferably, the video-based surgical targeting system further comprises 
instrument tracking means for tracking the position of a surgical instrument. 

In an alternative form of tiie invention, tiie video-based surgical targeting 
system comprises a patient-specific database comprising a plurality of 2-D 
images of the anatomical structure of a patient; marker placement means for 
inserting virtual planning markers into the 2-D images contained in the patient- 
specific database; an image generator for generating a virtual image of the 
anatomical structure represented by the plurality of 2-D images contained in 
the palient-q[>ecific database; real-time image generating means for genmting 
a real image of die anatomical stracture of a patient; video mixing means for 
mixing die virtual image and the real image into an output image, whereby the 
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ou^ut image may comprise either one of the two images exclusive of the 
other, or a composite of both images; and display means for displaying the 
output image. 

In another alternative form of the invention, the video-based surgical 
taigding ^tem comprises a patient-specific database comprising a plurality 
of images of the anatomical structure of a patient; a patient-specific 3-D 
computer model of the anatomical structure of the patient, the patient-specific 
3-D computer model being assembled from the plurality of images contained 
in the patient-specific database; marker placement means for (i) inserting 
virtual planning markers into the images contained in the patient-specific 
database, and/or (ii) adjusting the positions qf virtual planning markers 
inserted into tfie images contained in Ae patient-specific database and 
thereafter incorporated into the patient-specific 3-D computer model, or 
inserting virtual planning markers iato die 3-D computer model; an image 
generator for generating a virtual image of the anatomical structure modeled 
by die patient-spedfic 3-D computer model; real-time image generating means 
for generating a real image of the anatomical structure of a patient; video 
mixing means for mixing the virtual image and the real image into an ou^ut 
image, whereby the ou^ut image may comprise either one of the two images 
exclusive of the other, or a composite of botii images; and display means for 
displaying said output image. 
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Jnyet another alternative foim of die inventioii, the video-based surgical 
taigeting system comprises a patient-specific database comprising a plurality 
of 2-D images of the anatomical structure of a patient; marker placement 
meanis for inserting virtual plaiming markers into the 2-D images contained in 
die padent-spedfic database; an image generator for generating a virtual hnage 
of die anatomical structure defined by the patient-specific database; real-time 
image generating means for generating a real image of the anatomical structure 
of a patient, the real-time image generating means being adapted so as to 
dynamically update die patient-specific database via the real image; video 
mixing means for mixing die vutual image and die real image into an output 
image, v/bmby the output image may comprise eidier one of the two images 
exclusive of the other, or a composite of both images; and display means for 
displaying said output image. 

The present invention also comprises a method for targeting an anatomi cal 
structure using die video-based surgical targeting system. 

Brief Pescriptton Qf The Drawings 

These and odier objects and features of the presmt invention will be more 
fully disclosed or rendered obvious by the following detailed description of the 
prefened embodimoits of the invoition, which are to be considered together 
widi the accompanying drawings wherein like numbers refer to like parts and 



SUBSTITUTE SH^ (RULE 26) 



wo 96/10949 



PCTAIS95/13353 



-14- 



furtheir herein: 

Fig. 1 is a schematic view of the major components of a video-based 
smgical targeting system formed in accordance widi tfie present invention; 

Fig. 2 is a view of an memplaiy patient-specific 2-D image of tfie sort 
contained in the patient-specific database; 

Fig. 3 is a view of an exemplaiy patient-specific 2-D image of the sort 
contained in the patient-specific database, wherein the image has had several 
virtual planning markers placed into the image; 

Fig. 4 is a composite view of (i) a real image obtained by a real-time 
imaging device (e.g. a video camera), and (ii) a virtoal image generated fixmi 
the 3-D computer model, wherein the two images are not yet in registration 
wi& one another. 

Fig. S is a composite view of (i) a real image obtained by a real-time 
wttflging device (e.g. a video camera), and (ii) a virtual image generated from 
the 3-D computer model, i^erein die two images have been placed in 
registration with one another; 

Fig. 6 is a view like that of Fig. 5, except tiiat the virtual image generated 
firom the 3-D computer model has been sliced away to expose the virtual 
planning maikers; 

Fig. 7 is a view like that of Fig. 5, except Aat the virtual image generated 
from die 3-D computer model has been faded away to expose the virtual 
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planning madcers; 

Fig. 8 is a schematic view showing the podtional relationships between 
various elements of Ae video-based singical targeting system and die patient's 
anatomical stracture, as well as die positional relationship between an 
exemplary singical instalment and the patient's anatomical stracture; 

Fig. 9 is a virtual image generated from the 3-D computer model, showing 
a virtual "over the shoulder*' view from the tip of a tracked surgical instrument; 
and 

Fig. 10 is a flowchart illustrating one way of operating a video-based 
surgical targeting system formed m accordance with the present invention. 

Detailed Description Of The Preferred Embodiments l 

Looking first at Fig. 1» the starting point for the present invention involves 
a data acquisition device 5 which is adapted to generate patient-qiecific 2-D 
images. For example, data acquisition device 5 might comprise an MRI 
device, a CT scanner or any o&er scanning device of the sort adapted to 
provide a series of 2-D images of the patient* s anatomical structure^ where 
each 2-D image corresponds to a plane or slice taken through the scanned 
structure. Alternatively, data acquisition device 5 might comprise an X-ray 
machine or other imaging machine adapted to provide a set of 2-D images of 
Ihepatienfs anatomical structure, whore each2-D image corresponds to a data 
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set relating to the scanned structure. 

The patient-spedfic 2-D images generated by data acquisition device 5 are 
stored in a patient-specific database 10. Patient-specific database 10 is 
constructed so that the patient-spedfic 2-D images can be accessed 
individuaUy or in ai^ particular sequence desired. Preferably patient-specific 
database 10 comprises an appropriate set of data contained in a computer 
storage system. 

Ihe patient-specific 2-D images stored in patient-specific database 10 are 
used in building a patient-specific 3-D computer model IS. Patient-specific 
3-D computer model 15 may be built firom the information stored in 
patient-q>ecific database 10 using ai^ one of the many algorithms well known 
in the art, so long as the patient-specific 3-D computer model is capable of 
generating virtual images of the patienfs anatomical stracture firom 
substantially any location where a real camera may be positioned relative to 
that anatomical structure during a given procedure. Alternatively, patient- 
specific 3-D computer model 15 may be built from tiie infoimation stored in 
patient-specific database 10 using any one of the many algorithms well known 
in the art, where the patient-specific 3-D computer model is capable of 
generating virtual images of the patient's anatomical structure from a 
predetermined set of possible virtual camera positions. Preferably, 
patient-specific 3-D computer model 15 is constructed so that it is csqpable of 
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genenting vixtual images of the patient's anatomical structure from any 
location whatsoever, with any desired field of view, and including any otfier 
camexa-qiecific oiteria desired (e.g. particular camera characteristics such as 
focus, optical characteristics, optical distortions, etc.). By way of example 
one possible algoridmi for building die patient-specific 3-D computer model 
out of the information stored in the patient-specific database 10 might be the 
Marching Cubes algorithm. 

In one preferred embodiment of the invention, patient-specific 3-D 
computer model IS comprises an appropriate computer sofiware model 
resident on an appropriate digital computer. Patient-specific 3-D computer 
model 15 is preferably constructed using a plurality of polygonal models to 
model die patienf s real anatomical structure. As is well known in the art, such 
polygonal models generally comprise a collection of points defining the 
sut£u:e of the 3-D conqmter modd, along with some connectivity information 
relating to how these surface points are coimected to one anodier. Preferably 
there is one polygonal model per anatomical structure being modeled. Of 
course, the choice of which particular computer software model is used to 
model a particular anatomical structure ultimately depends on a variety of 
factors, such as the particular anatomical structure being modeled, the 
computer hardware available, the volume of data to be handled, the particular 
medical procedure which is to be conducted, etc. 
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Of course, it is also anticipated that tiie patient-specific 3-D computer 
model could be dynamic as well as static. For example, dynamic changes 
could occur in Ike model as a result of computer simulation (e.g, through die 
use of a Finite Element Model) or because more current real-time data (e.g. 
from video camera 45) is used to iq)date the model, or bofli. 

It is also anticipated diat patient-specific 3-D computer model 15 mi^t 
include data from sources other than the 2-D images contained in 
patient-specific database 10. For example, patient-specific 3-D computer 
model 15 migiht include information obtained from a 3-D surface digitizer such 
as that used in certain types of cranial suiigexy,^ or patient-specific 3-P 
computer model 15 might include information obtained from a temperature 
probe or a chemical probe. In fact, in addition to the 2-D image data obtained 
from patient-specific database 10, patient-specific 3-D computer model 15 
might include substantially any type of information gathered by almost any 
type of device or sensor or process. 

An integral part of the present invention involves the placement of virtual 
planning markers into the patient-specific data, using apparatus 20. This may 
be done by placing such virtual plarming markers into one or more of the 
patient-specific 2-D images contained in patient-specific database 10 before 
patient-spedfic 3-D compute model 15 is created; or by placing such virtual 
planning markers into patient-specific 3-D computer model 15 after that model 
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has been created; or by some CQmbination of flie two. 

Nfoepardculady; apparatus 20 prefeiably comprises computer hardware 
and software adq>ted to allow a physician to access one or more of the 
patient-specific 2-D images contained in patient-specific database 10 and 
present them f(»rviewing» in the manner shown in Fig. 2. Then, using a mouse 
or other data entry device, the physician can place one or more virtual 
plaiming maricers 25 into an accessed patient-specific 2-D image, in die 
manner shown in Fig. 3. These virtual plaiming markers 25 can consist of 
substantially any geometric form such as a point, a line, a circle, a plane, a 
path (either straight or curvedX etc., and are positioned about anatomical 
structures ofparticular interest to the physician. For example, virtual plaiming 
markers 25 might be placed about a suspected tumor, or they might be placed 
about particulaily sensitive anatomical structures such as vascular tissue or 
nerve bundles, etc. Naturally, the particular geometric form chosen for a 
particular virtual plannii^ maikermay depend on a variety of factors, such as 
tfie particular anatomical structure involved, die computer hardware available, 
the volume of data to be handled, tiie particular medical procedure to be 
conducted, etc. Regardless of the particular form of virtual planning marker 
chosen, each virtual planning marker 25 which is placed on the 2-D image 
using apparatus 20 is then stored in patient-specific database 10 along with its 
associated 2-D image. As a result i^en patient-specific 3-D computer model 
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IS is subsequently created fiom the data contained in patient-specific database 
10, patient-q)ecific 3-D computer model 15 will incorporate virtual planning 
maikers 25 into tilie model. 

Ahematively, a pl^rsician may use apparatus 20 to access patient-specific 
3-D computer model 15 and tiien, using a mouse or odier data entiy device, 
adjust the position of one or more of the virtual planning maikers 25 which 
were previously placed into patient-specific database 10 (and thus 
incorporated into patient-specific 3-D computer model 15 when that 3-D 
computo^ model was oeated). The adjusted positions ofdiese virtual planning 
markers 25 are then stored in patient-specific 3-D computer model 15. 
Furthermore, an updated 2-D inuige of these virtual plaiming markers can then 
be incorporated into the patient-specific database 10. ? 

Furthermore, where patient-specific 3-D computer model 15 has not 
already had a particular virtual plaiming marker 25 incorporated therein, a 
physician may also use apparatus 20 to access patient-specific 3-D computer 
model 15, and tiien use a mouse or other data entry device to insert one or 
more virtual planning markers 25 directiy into patient-specific 3-D computer 
model 15. 

Once virtual planning markers 25 have been properly positioned m 
patient-specific 3-D computer model 15, virtual images incorporating these 
virtual planning markers can be created by an image generator 30, fed through 
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a video mixing device 35, and then presented to the physician on display 40. 

At the same time, real iniages obtained fiom a video camera 45 can be fed 
dnoa^ vi^ mixing device 35 and then presented to tiie physician on display 
40. 

More paiticulaily, image generator 30 comprises compute hardware and 
software of the sort well known in &e art for generating viitual images from 
patimt-specific 3-D computer model 15. 

Video mixing device 35 , comprises a video mixing device of the sort well 
known in die ar^ v^ereby the surgeon can selectively display (i) a virtual 
image created by image generator 30 from patient-specific 3-D computer 
model 15, (n) a real image obtained from video camera 45, or (iii) a composite 
of die virtual image and die real image, where the virtual image is 
superimposed against the real image. 

In order for die foregoing composite image to be more useful to die 
physician, die virtual image generated from patient-specific 3-D computer 
model 15 should be placed into registraticm widi the real image obtained from 
video camera 45. This is done in several steps. 

First, and looking now at Fig. 4, a virtual image 50 is created by image 
generator 30, fed through video mixing device 35, and presented on display 
40. Simultaneously, a real image 55 is captured by video camera 45, fed 
dirough video mixing device 35, and presented on display 40. Video mixing 
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device 35 is arranged so as to present virtual image 50 and real image' 55 
simultaneously on display 40, widi die virtual image being superimposed on 
the real image (Le., so that die virtual image is in the foreground and the real 
image is in die background, in the maimer sho^ in Fig. 4). 

Second, and looking now at Fig. 5, the two images are placed into proper 
registration witfi one anodier. This image registradon can be accomplished in 
eidier one of two ways. 

A first, and generally more preferable, technique involves holding the 
position of video camo'a 45 (and hence real image 55) constant and moving 
the position of the "virtual object" or the "virtual camera" by means of 
apparatus 57 until the virtual image 50 is brought into registration with real 
image 55. This can be done automatically by having image generator 30 use 
a search algoridun to match the virtual image to the real image, in which case 
^aratus 57 includes computer hardware and software of the sort well known 
in die art to cause image generator 30 to work du*ough a search algorithm to 
match the virtual image to die real image; or it can be done manually, in which 
case apparatus 57 includes computer hardware and software of the sort well 
known in the art to allow die physician to drag the virtual image 50 into 
registration widi real image 55, using a mouse or other data entry device 
attached to image generator 30. One convenient way that mouse motion can 
be used to control die 3-D movements of the patient-specific model is to map 
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SDchmotioii into vectors defined by the view direction of tiie virtual camera, 
as will be well known to persons dolled in die art. 

A second, and generally less preferable, technique involves holding the 
virtual image SO constant and moving video camera 45 (or the patient) until 
real image 55 matches virtual image 50. 

Regardless of vdiich technique is used, once virtual image 50 has been 
matdied to real image 55, die position of die "virtual camera" (i.e., the location 
fiom wfaidi the virtual image appears to be seen) will be matched to the actual 
position of video camera 45. 

Once virtual image 50 has been placed into proper registration with real 
image 55, image generator 30, video camera 45 and video mixing device 35 ^ 
can be used to present the virtual and real images on display 40 in various ' 
^ presentation formats so as to facilitate a particular medical procedure. In 
particular, one can use image generator 30, video camera 45 and video mixing 
device 35 to siq>erimpose a virtual image (genmted fi^om patient-specific 3-D 
computermodel 15)againstarealimage(generatedby video camera 45), with 
image generator 30 being directed to modify the virtual image so as to e>q>ose 
one or more of the virtual planning markers 25 present in patient-specific 3-D 
computer model 15, whereby tfie anatomy highlighted by virtual planning 
markers 25 will be brought quickly to tiie attention of the physiciaiL 

In this respect it is to be appreciated tiiat since virtual planning markers 
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25 will fiequen% be located widiin interior portions of the structure modeled 
by patient-specific 3-D computer model IS, exposing the otherwise-hidden 
virtual planning mailcers to view will involve rendering some or all of die 
virtual anatomical stracturetFBnq»rent or semi-tnmsparent. This can be done 
either by (i) slicing away any portions of die virtual image required in order 
to aqx>se virtual planning maikers 25, whoeby diose virtual plaiming markers 
will be rendered visible against the background real image, in die manner 
shown in Fig. 6; or (ii) fading away some or all of the virtual image so as to 
expose such virtual plaiming markers against the real image, in the maimer 
shown in Fig. 7. As a result, a physician can then conduct a medical 
procedure with die confidence of using real images generated by video camera 
45, while having virtual planning markers 25 superimposed against the real I 
image of die anatomical structure so as to help guide die procedure. 

While die foregoing system constitutes a major improvraient over the 
prior art, it has also been recognized that it can itself be improved upon 
significandy by adding camera tracking means to video camera 45. 

More particularly, in many medical procedures, the position of video 
camera 45 may change on a fairly frequent basis. Since it is important for 
virtual ima^ 50 (generated from patient-specific 3-D computer model 15) to 
remain in proper registration widi real image 55 (obtained from video camera 
45), it is necessary for the video-based surgical targeting system to reestablish 
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proper correspondence between tfie two images each time the video camera 
moves. As noted above» this proper correspondence can be reestablished each 
tune video camera 45 moves, by eilha (i) having the physician manually drag 
virtual image 50 into registration widi real image 55, using a mouse or otfier 
data entiy device attached to ima^ generator 30, or (ii) having the system use 
a search algoriftm to match the virtual image to the real image. While both of 
these procedures are generally capable (assuming tiiat the anatomical structure 
remains stationary) of accurately yielding die desired image correspondence, 
eadi one suffers firom a significant disadvantage. In the first case, requiring 
the pl^dan to physically drag the virtual image into registration witfi the real 
image each time the video camera moves can be inconvenient where the video 
camera is moved about firequendy. In die second case, it can take the system 
a substantial amount of time to solve for the correct image correspondence 
when using a search algoritfun, even where high speed hardware is utilized. 
Even where speed is not a problem, automated search strategies of the type 
generally used are prone to errors through false matches. 

Accordingly, it has been recognized that by putting a tracker on video 
camera 45, die position of die real camera can be continuously monitored, 
whereby each time die real camera moves, the position of the virtual camera 
can be correspondingly adjusted by image generator 30. In this way, the 
virtual image generated by image generator 30 can quickly and easily be 



SUBSTITUTE SHEET (RULE 2S) 



wo 96/10949 



PCT/DS95/13353 



-26- 



mamtained in legistratioii widi die real image, regardless of how often video 
camera 45 is moved. 

More particularly, and looking now at Fig. 8, there is shown a video 
camera 45, an anatomical stnictiire 60, and a tracker system 65. Tracker 
system 65 comprises a tracker 70 which is attached to video camera 45, and 
a tracker base 75 which defines the coordinate system of the tracker system. 

In this setting, Mpc can be considered to represent the matrix 
transformation from the patient's anatomical structure 60 to camera 45; 
can be considered to rq;>resent the matrix transfomiation from camera 45 to 
tracks base 75; and Mpp can be considered to present die matrix 
transfomiation from anatomical stracture 60 to tracker base 75. 

is known from the tracker system. Furthermore, once the virtual 
image generated by image generator 30 has been placed in registration with the 
real image generated by camera 45, the virtual camera position will be known 
relative to the virtual anatomical stmcture, and hence the real camera position 
win be known relative to die real anatomical structure. Thus, real matrix Mpc 
will also be known. In addition, since Mct and Mpc are then both known, it is 
possible to solve for Mp^. Accordingly, the position of anatomical structure 
60 will dien also be known within the relative coordinate system defined by 
die tracker system. 

In view of the foregoing, the virtual image generated by image generator 
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30 can be quickly and easity maintained in registration with tfie real image, 
regardless of how often camera 45 is moved. 

FmthermoiCy anodier tradcer 80 can be positioned on a surgical instrument 
85. In diis case, tracker 80 wiU provide M,,, where Mn represents die matrix 
transfomiadonfiQmtraclKr base 75 to instrument 85. Accordingly, once Mn- 
and Mti are known, it is possible to solve for Mp,, where Mp, represents flie 
matrix transformation from die patient's anatomical structure 60 to instrument 
85. With Mpi known, it is then possible to track the position of surgical 
iiistrurnent 85 relative to aiiatoinical structure 60. This is a very powerful tool, 
since it allows a virtual rqiresentation of surgical instrument 85 to be 
accurately added to the properly registered virtual and real images presented 
on display 40, i.e., it allows a virtual representation of instrument 85 to be 
shown in proper registration with a real image of anatomical structure 60 on 
display 40. 

Furthermore, as the position of surgical instrument 85 is tracked relative 
to anatomical stracture 60, it is also possible to provide a virtual image of 
surgical instrument 85 as that surgical instrument 85 moves through the 
anatomical structure, even whm some or all of that instrument might be 
hidden from the view of video camera 45. For example, and looking now at 
Fig. 9, it is possible to generate a virtual "over die shoulder^ view of the distal 
tip of surgical instrument 85 moving through anatomical structure 60, where 
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that tip will be shown in proper correspondence to various structures, e.g. a 
tumor 95, vascular structures 100, etc. It is also possible to generate an 
additional virtual object 90 showing a line extended along the axis of the 
surgical instrument 85 so that the physician can see where the surgical 
instrument would go if moved fiuther along its current trajectory. Of course, 
since Ae virtual image generated by the video-based surgical targeting system 
will then differ significantly from the real image generated by video camera 
45, video mixing device 35 should be directed to totally suppress the real 
image generated by video camera 45 so that it will no longer be shown on 
displiqr 40, in order to avoid confusing the physician. 

Modificarion g Of The Preferred Embodiments 

It is, of course, possible to modify the preferred embodiments disclosed 
above without departing from the scope of the present invention. 

Thus, for example, while only one camera 45 has been disclosed above, 
it should be appreciated that two or more cameras 45 m^ be used. 
Furthennore, it should also be appreciated that camera 45 may comprise a 
video camera, or it may comprise an endoscope, or it may comprise some 
odier type of real-time inuige capturing means, e.g. it may comprise an 
ultrasound device. 

Additionally, it should be recognized that the system might be adapted so 
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tbat patient-specific database 10 comprises iioii-2-D images, e.g. database 10 
might comprise one or more images generated by a Cyberware 3-D scanner. 

Alternatively, patient-spedfic 3-D computer model 15 might be omitted 
fiom the fq[^mratiis entirety. In tiiiscase» the virtual iinages could be generated 
fiom patient-qiecific database 10 by image generator 30 through the use of a 
volume rendering procedure of the sort well known in the art. 

It is also anticipated that one could use a tracked surgical instrument 85 
to determine the location of anatomical structure 60. This can be 
accomplished using tiie tracked surgical instrument 85 to engage known 
fiducial points on tiie anatomical structure. Alternatively, a tracked surgical 
instrument 85 can be used to sample multiple surface points located on the 
anatomical structure and then use a data matching procedure to correlate the 
san9>led points wifli either patient-specific database 10 or patient-specific 3-D 
computer model 15. By way of example, a least squares fit might be used to 
correlate the sampled points witii the 3-D computer model. 

Also, it should be appreciated tiiat tiie video-based targeting system could 
be used to taiget objects in non-medical qjplications, e.g. it could be used to 
target objects concealed widun the interior of complex machines, or objects 
(e.g. a conduit) concealed beneath surface of a structure (e.g. the floor of a 
building). 

Furthermore, it is possible to place tracking means directiy on the patient 
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so as to track the position of the patient's anatomical stracture. 
Still other changes will be obvious to a person skilled in the art. 

Advantages Qf The Present tovention 

Numerous advantages are obtained through die use of tiie present 
invention. 

For one diing, a surgical targeting system is provided to facilitate 
locating a particular anatomical structure during a medical procedure. 

For anodier diing, a video-based surgical targeting system is provided 
to facilitate locating a particular anatomical structure during a medical 
procedure. I 

And a video-based surgical targeting system is provided which 
permits a series of patient-specific 2-D images (obtained by scanning 
patient anatomy using one or more scanning devices of the type described 
above) to be assembled into a 3-D computer model of the patient's scanned 
structure. 

Also, a video-based surgical targeting system is provided which allows 
a physician to view the aforementioned patient-specific 2-D images on a 
display in any desired access sequence. 

Furthermore, a video-based surgical targeting system is provided 
which allows a physician to assemble a series of patient-specific 2-D 
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images into a patient-specific database, and tiien to genmte virtual images 
fix>m the aforementioned patient-specific database, as seen from any desired 
virtual camera position, for viewing on a display. 

And a video-based surgical targeting system is provided which allows 
a physician to generate virtual images firom die aforementioned 3-D 
computer model, as seen from any desired virtual camera position, for 
viewing on a display. 

In addition, a video-based surgical targeting system is provided which 
permits a physician to place virtual plarming markers about any sites of 
interest while viewing one or more of the aforementioned patimt-specific 

2- D images, with those virtual plarming markers then being incorporated 
into the 3-D computer model, whereby those virtual plarming maricers can 
be displayed in their appropriate 3-D positions when generating virtual 
images of the 3-D computer model. 

And a video-based surgical targeting system is provided which permits 
a physician to place virtual planning markers about any sites of interest 
while viewing virtual images of the 3-D computer model, with those virtual 
plarming markers tiien being incorporated into the 3-D computer model, 
whereby tiiose virtual plarming markers can be displayed in their 
appropriate positions when subsequently generating virtual images of die 

3- D computer model. 
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And a video-based surgical taigeting system is provided which pmnits 
a physician to place virtual planning maikers about any sites of interest 
while viewing virtual images of die 3-D computer model, with those virtual 
planning maikers then being incorporated into the 3-D computer model and 
into the database of 2-D images, whoneby those virtual planning nuukers 
can be di^layed in dieir appropriate positions when subsequently 
generating virtual images of die 3-D computer model or when subsequendy 
displ^ring 2-D images from die patient-specific database. 

For anodier thing, a video-based surgical targeting system is provided 
which permits a real inuige obtained by a real-time imaging device (e.g. a 
video camera) to be displayed to a physician, and which permits a virtual 
image generated from die 3-D computer model to be displayed to a 
physician, according to the directive of the physician. 

And a video-based surgical targeting system is provided which permits 
a real image obtained by a real-time imaging device (e.g. a video camera) to 
be appropriately meiged with a corresponding virtual image generated from 
the 3-D computer model. 

Also, a video-based surgical targeting system is provided which 
permits a real image obtained by a real-time imaging device (e.g. a video 
cam^) to be ^propriately meiged with a corresponding virtual image 
generated from the 3-D computer model, whereby the two images will be in 
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registration widi one another. 

Furthermore, a video-based surgical targeting syston is provided 
which permits a real image obtained by areal-time imaging device (e.g. a 
video camera) to be merged with a corresponding virtual image gmoated 
firom the 3-D computer model, whereby die two images will be in 
registration widi one another, and whereby the physician can choose to 
display either one of the two images exclusive of die other, or a composite 
of both images simultaneously. 

And a video-based surgical targeting system is provided which permits 
a real image obtained by a real-time imaging device (e.g. a video camera) to 
be merged widi a corresponding virtual image generated from the 3-D 
computer model, whereby the two images will be simultaneously displayed 
in registration widi one another, and whereby the physician can modify die 
virtual image generated from die 3-D computer model as needed, by 
cUpping or fiiding, so as to expose the virtual plarmmg iiiarkers to view, 
with die virtual plaiming markers being superimposed on the real image 
generated by the real-time imaging device. 

And a video-based surgical targeting system is provided which permits 
a real image generated by a real-time imaging device (e.g. a video camera) 
to be mofged with a corresponding virtual image generated from the 3-D 
computer model, whereby the two images will be simultaneously displayed 
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in fegtstration with one another, and whereby the physician can modify the 
virtual image generated from the 3-D computer model so as to e;q>ose only 
the virtual planning markers to view, with the virtual planning markers 
being siq)erimposed on the real image generated by die real-tune imaging 
device. 

Moreover, a video-based surgical targeting system is provided which 
pemiits a real image obtained by a real-time imaging device (e.g. a video 
camera) to be merged with a corresponding virtual image generated from 
the 3-D computer model, whmby die two images will be simultaneously 
displayed in registration with one another, and whereby this registration 
will be automatically maintained even as die real-time imaging device is 
moved about relative to the anatomical site, with the virtual image being 
automatically generated so as to follow the real image. 

And a video-based surgical targeting system is provided which permits 
a real image obtained by a real-time imaging device (e.g. a video camera) to 
be merged with a corresponding virtual image generated from the 3-D 
computer model, whereby the two images will be simultaneously displayed 
in registration with one anodier, and whereby this registration will be 
automatically maintained dirough the use of a computerized position and 
orientation tracker connected to die imaging device even as the real-time 
imaging device is moved about relative to the anatomical site, with the 
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virtual image being automatically generated so as to follow the real image. 

Also, a video-based surgical targeting system is provided which 
permits a real image obtained by a ieal*time imaging device (e.g. a video 
camera) to be merged vnA a corresponding virtual image generated firom 
the 3-D computer model, whereby tiie two images will be simultaneously 
displayed in registration widi one another, and whereby diis registration 
will be automatically maintained through the use of a computer search 
algorithm based on the real image and the virtual image even as the 
real-time imaging device is moved about relative to the anatomical site, 
with the virtual image being automatically generated so as to follow the real 
image. 

And a video-based surgical targeting system is provided which permits 
a real image obtained by a real-time imaging device (e.g. a video camera) to . . 
be meiged with a corresponding virtual image generated from the 3-D 
computer model, whereby the two images will be simultaneously displayed 
in registration with one anodier, and whereby a surgical instrument can be 
tracked along with the real-time imaging device in order that the surgical 
instrument can be guided about die anatomical site using the virtual image 
generated from die 3-D computer model. 

In addition, a video-based surgical targeting system is provided which 
permits a virtual image to be generated from the 3-D computer model, and 
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fiirtiier wherein diis virtual image can correspond to an "over the shoulder*' 
view of die working tip of a surgical instrument located at die anatomical 
site. 

And an improved method for locating anatomical structures durmg a 
medical procedure is provided. 



I 
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What Is Claimed Ts: 

L A video-based surgical targeting system com priCTng ' 

a patient-specific database comprising a plurality of 2-D images of the 
anatomical structure of a patient 

a patient-specific 3-D computer model of said anatomical structure of 
said patient, said patient-specific 3-D computer model being assembled from 
said plurality of 2-D images contained in said patient-specific database; 

marker placement means for (i) inserting virtual planning markers into 
said 2-D images contained in said patient-specific database, and/or (ii) - 
adjusting the positions of virtual planning markers inserted into said 2-D 
images contained in said patient-specific database and diereafter incoiporated 
into said patient-qiecific 3-D computer model, or inserting virtual planning 
markers into said 3-D computer model; 

an image generator for generating a virtual image of the anatomical 
structure modeled by said patient-specific 3-D computer model; 

real-time image generating means for generating a real image of the 
anatomical stracture of a patient; 

video mixing means for mixing said virtual inuige and said real image 
into an output image, whereby said output image may comprise either one of 
die two images exclusive of the other, or a composite of both images; and 
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display means for displaying said output image. 

2. A video-based suigical targeting system according to claim 1 wherein 
said system fistiier conqnises r^istration means for placing said virtual image 
in registration with said real image. 

3. A video-based surgical targeting system according to claim 2 wherein 
said registration means include means for manually aligning said virtual image 
with said real image. 

4. A video-based surgical targeting system according to claim 2 wherein 
said registration means include means for automatically aligning said virtual 
image witfi said real image. 

5. A video-based surgical targeting system according to claim 2 wherein 
said registration means include tracking means for traddng the position of said 
real-time image generating means. 

6. A video-based surgical targeting system according to claim 2 wherein 
said system furdier comprises instrument tracking means for tracking the 
position of a surgical instrument 
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7. Avide(>-basedsi]i:gicdtaigetingsy5tm 

said system fiiidier comprises patient traddng means for traddng the position 
of said anatomical stmcture. 

8. A video-based smgicaltaigeting system according to claim 1 wli»ein 
said real-time image generating means comprise a video camera. 

9. A video-based singical taigeting system according to claim 1 wherein 
said real-time image generating means comprise an endoscope. 

10. A video-based surgical targeting ^stem according to claim 1 
v^ierein said real-time image generating means comprise an ultrasomid device. 

11. A video-based surgical targeting system according to claim 1 
wherein said patient-specific 3-D conqiuter model is constructed out of one or 
more polygonal models. 

12. A video-based surgical targeting system according to claim 1 
wherein said image generator generates said virhial image by a polygonal 
rendering process. 
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13. A video-based sm^gical targeting system comprising: 

a patient-qiecific database comprising a plurality of 2*D images of the 
anatomical structure of a patient; 

marker placement means for inserting virtual plarming markers into said 
2-D images contained in said patient-spedfic database; 

an image generator for generating a virtual image of the anatomical 
structure represented by said plurality of 2-D images contained in said patient- 
specific database; 

real-time image generating means for generating a real image of the 
aimtomical stracture of a patient; 

video mixing means for mixing said virtual image and said real image 
into an output image, whereby said output image may comprise either one of 
tfie two images exclusive of the other, or a composite of both images; and 

display means for displayirig said output image. 

14. A video-based surgical targeting system according to claim 13 
wherein said image generator generates said virtual image by a volume 
rendering process. 

15. A video-based surgical targeting system comprising: 

a patient-specific database comprising a plurality of images of the 
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anatomical strocture of a patioit; 

a patient-specific 3-D conqniter model of said anatomical strocture of 
said patient, said patient-specific 3-D computer model being assembled from 
said plurality of images contained in said patient-specific database; 

madcer placement means for (i) inserting virtual planning mailcers into 
said images contained in said patient-specific database, and/or (ii) adjusting 
the positions of virtual planning markers inserted into said images contained 
in said patient-specific database and thereafter incorporated into said patient- 
specific 3-D computer model, or inserting virtual planning markers into said 
3-D computer model; 

an image generator for generating a virtual image of the anatomical 
structure modeled by said patient-specific 3-D computer model; 

real-time image generating means for generating a real image of the 
anatomical structure of a patient; 

video mixing means for mixing said virtual image and said real image 
into an output image, whereby said output image may comprise either one of 
the two images exchisive of the odier, or a composite of both images; and 

display means for di^U^dng said output image. 

16. A video-based suigical targeting system comprising: 

a patient-specific database comprising a plurality of 2-D images of the 
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anatomical stnictme of a patient; 

madcer placement means for insoting virtual planning maikers into said 
2-D images contained in said patient-specific database; 

an image generator for gmerating a virtual image of the anatomical 
structure defined by said patient-specific database; 

real-time image generating means for generating a real image of the 
anatomical stnicture of a patient^ said real-tune image generating means being 
adapted so as to dynamically update said patient-specific database with said 
real image; 

video mixing means for mixing said virtual image and said real image 
into an output image, whereby said output image mdy comprise either one of 
the two images exclusive of the odier, or a composite^of both images; and 

display means for displaying said output image. 

17. A video-based surgical targeting system according to claim 2 
wherein said registration means comprise a surgical instrument and tracking 
means for traddng the position of said surgical instrument, and fintiier wherein 
said registration means comprise means for effecting registration when said 
sur;^cal instrument engages known fiducial marks located on said anatomical 
structure. 
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18. Aiddeo4)asedsuigtcaltiack]iigq^^^ 

said registratioii means conqnise a suigical instniment and tracking means for 
tracking Ae position of said surgical instrmnent, and further wherein said 
registration means comprise means for effecting registration when said 
suigical instrument samples multiple sur&ce points located on said anatomical 
structure, with said means for effecting registration using a data matching 
procedure to condate said sampled points widi said patient-specific database. 

19. A \ddeo-based suigical tracking system according to claim 4 wherein 
said registration means are adapted to use said real image to track and update 
said patient-specific 3-D computer model 

20. A method for targeting an anatomical structure, said method 
comprising the steps of: 

(1) providing a video-based suigical targeting system comprising: 
a patient-specific database comprising a plurality of 2-D images of 
the anatomical structure of a patient; 

a patient-specific 3-D computer model of said anatomical stmcture 
of said patient said patient-specific 3-D computer model being assembled 
fiom said plurality of 2-D images contained in said patient-specific database; 
marker placemrat means for (i) inserting virtual planning markers 
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into said 2-D images contained in said patient-specific database, and/or (ii) 
adjusting tiie positions of virtual plannii^ markers inserted into said 2-D 
images contained in said patient-specific database and thereafter incorporated 
into said patient-spedfic 3-D computer model, or inserting virtual planning 
madcers into said 3-D computer model; 

an imagp generator for generating a virtual image of tiie anatomical 
structure modeled by said patient-q)ecific 3-D computer model; 

real-time image generating means for generating a real image of the 
anatomical structure of a patient; 

video mixing means for mixing said virtual image and said real image 
into an output image, whereby said output image may comprise either one of ^ 
the two images exclusive of tiie other, or a composite of both images; and ^ 

display means for displaying said output image; 

(2) inserting a visual planning marker into a 2-D image contained in said 
patient-specific database; 

(3) gmfratmg a Virtual image of die anatomical structure defined by said 
patient-specific 3-D model, and generating a real image of tfie anatomical 
structure of a patient; 

(4) mixing said virtual image and said real image into an output image, 
whereby said output image comprises a composite of botii images, and 
displaying said output image on said display means, wherein said virtual image 
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and said real image are mixed so tfiat said visual plamiing maiker i$ eTqposed 
to view so as to be superimposed against said real image. 

21. A method acconiing to claim 20 wherein said virtual in^ 
real image are mixed so as to be in registration widi one anodier. 

22. A method according to claim 21 wherein said video-based surgical 
targeting system con^rises means for maintaining said virtual image and said 
real image in registration with one anodier even when said real-time image 
generating means move relative to said anatomical structure. 

23. A method for targeting an anatomical stnicture, said method 
comprising the steps of: 

(1) providiiig a video-based surgical targeting system comprising: 
a patient-specific database comprising a plurality of 2-D images of 
the anatomical structure of a patient; 

marker placement means for inserting virtual planning markers into 
said 2-D images contained in said patient-specific database; 

an image generator for generating a virtual image of the anatomical 
structure represented by said plurality of 2-D images contained in said patient- 
specific database; 
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real-time ima^ generating means for generating a real image of die 
anatomical structure of a patient; 

video mixiiig means for mixing said virtual image and said real image 
into an output image, whereby said ou^ut image may comprise either one of 
the two images exclusive of die other, or a composite of bodi images; and 

display means for displaying said output image; 

(2) inserting a visual phmning marker into a 2-D image contained in said 
patient-specific database; 

(3) generating a virtual image of the anatomical structure represented by 
said plurality of 2-D images contained in said patient-specific database, and 
generating a real image of the anatomical structure of a patient; and 

(4) mixing said virtual image and said real image into an output image, 
whaeby said output image comprises a composite of both images, and, 
displaying said ou^mt image on said displ^ means, wherein said virtual image 
and said real image are mixed so diat said visual planning marker is e^qposed 
to view so as to be superimposed against said real image. 

24. A method for targeting an anatomical structure, said method 
comprising die stqps of: 

(1) providing a video-based surgical targeting system comprising: 
a patient-specific database comprising a plurality of images of die 
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anatomical structure of a patient; 

a patient-specific 3-D conq)uter model of said anatomical stracture 
of said patient, said patient-q)ecific 3-D computer model being assembled 
fixmi said plurality of images contained in said patient*specific database; 

marker placement means for (i) inserting virtual planning marken 
into said images contained in said patient-specific database, and/or (ii) 
adjusting the positions of virtual planning markers inserted into said images 
contained in said patient-specific database and diereafier incorporated into said 
patient-specific 3-D computer model, or inserting viitual planning markers into 
said 3-D computer model; 

an image generator for generating a virtual image of the anatomical 
structure modeled by said patient-specific 3-D computer model; 

real-dme image ^crating means for generating a real image of the 
anatomical structure of a patient; 

video mixing means for mixing said virtual image and said real image 
into an ou^ut image, whereby said output image may comprise either one of 
die two images exclusive of the other, or a composite of both images; and 

display means for displaying said output image; 

(2) inserting a visual planning marker into an image contained in said 
patient-specific database; 

(3) generating a virtual image of the anatomical structure defined by said 
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patient-specific 3-D model, and generating a real image of the anatomical 
structure of a patient; and 

(4) mixing said virtual image and said real image into an output image, 
whereby said ou^ut image comprises a composite of both images, and 
diq>layii% said ou^iut ima^ on said diqylay means^ wh^^ 
and said real image are mixed so that said visual planning maiker is exposed 
to view so as to bcf superimposed against said real image. 

25. A method for targeting an anatomical stracture, said method 
comprising the steps of: 

(1) providing a video-based surgical targeting system comprising: < 

a patient-specific database comprising a plurality of 2-D images of 
the anatomical structure of a patient; 

maricer placement means for inserting virtual planning markers into 
said 2-D images contained in said database; 

an image generator for generating a virtual image of the anatomical 
structure defined by said patient-specific database; 

real-time image generating means for generating a real image of the 
anatomical structure of a patient said real-time image generating means being 
adapted so as to dynamically update said patient-specific database with said 
real image; 
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video mixing means fomixing said virtue 
into an output image, whereby said oulpnt image may comprise either one of 
the two images exclusive of the other, or a composite of boA images; and 

dispUQT means for dispk^ing said output image; 

(2) inserting a visual plannmg marker into a 2-D image contained in said 
padent-spedfic database; 

(3) generating a virtual image of die anatomical structure defined by said 
patient-specific 3-D model, and generating a real image of the anatomical 
structure of a patient and 

(4) mixing said virtual image and said real image imo an output image, 
viiereby said ou^rat image comprises a composite of both images, and 
displaying said ou^ image on said display means, wherdn said virtual image 
and said real image are mixed so that said visual planning markor is ejqiosed 
to view so as to be superimposed against said real image. 
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